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Abstract 

Super- resolution is an important but difficult problem in image/ video 
processing. If a video sequence or some training set other than the given 
low- resolution image is available, this kind of extra information can greatly 
aid in the reconstruction of the high-resolution image. The problem is 
substantially more difficult with only a single low-resolution image on 
hand. The image reconstruction methods designed primarily for denois- 
ing is insufficient for super-resolution problem in the sense that it tends to 
oversmooth images with essentially no noise. We propose a new adaptive 
linear interpolation method based on variational method and inspired by 
local linear embedding (LLE). The experimental result shows that our 
method avoids the problem of oversmoothing and preserves image struc- 
tures well. 
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1 Introduction 

Super-resolution is an image processing technique to obtain high-resolution im- 
age from its low-resolution counterpart, either using information from a single 
image or image sequence. Most of the literature focus on the latter [5][8j. With 
only one image, it is generally difficult to interpolate the missing pixels with 
little prior information about the image. 

Many approaches have been proposed in recent years. For video/image 
sequence, information on subpixel motion between temporally adjacent im- 
ages can be utilized. Due to the ill-posedness of the super-resolution problem, 
regularization[5j or a Bayesian framework [9] usually needs to be set up, the two 
approaches being closely related. 

For only one single image, traditionally the most often used method is bilin- 
ear or bicubic interpolation, which works under the assumption that images are 
smooth for the most part. One recent advance on single image super-resolution 
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is the work by Freeman et al[3]. Using a large database of image patches con- 
sisting of low-resolution and high-resolution pairs, the algorithm will search 
the database for the corresponding high-resolution patch, for each of the image 
patches in the given low-resolution image. One obvious disadvantage of this 
approach is that it requires the existence of a database in the first place. 

We propose a novel algorithm for single image super-resolution, without the 
need for a database or other training images. Our algorithm is inspired by Local 
Linear Embedding (LLE) used for dimension reduction. By trying to express 
each pixel value as a linear combination of neighboring pixel values, we can 
estimate the local structure of the low-resolution image which we can then use 
to construct a filter for interpolation in the second step. 

One work similar in spirit to ours is [3j. There the authors also try to 
express the center pixel as a linear combination of its neighbors. Similar to 
Freeman [4 , their algorithm also requires a training set of pairs of patches in 
order to estimate those linear coefficients. And the coefficients are learned in 
the regression framework, giving as a result a spatially adaptive filter. There 
algorithm can be regarded as a complement to [4J which works well on cartoon 
images. 

One approach to single image super-resolution without training, which is 
also related to ours, is mentioned in [2] using digital TV filter which is the 
discrete counterpart of the variational functional method with a total variation 
(TV) norm. The digital TV filter is originally designed with the goal of im- 
age denoising in mind. Although the authors demonstrated that using a TV 
smoothing term can largely preserve edges and other important features, the 
result still looks blurred and blocky (using higher order smoothness term can 
alleviate the second problem with the cost of exacerbating the first problem). 
We think that directly using denoising algorithm on the super-resolution prob- 
lem is somewhat unnatural, with no special attention to the problem at hand. 
Our algorithm uses variational approach on the linear coefficients instead, so 
the smoothing term has an effect on the local filter, not directly on the image. 

2 local structure estimation 

Denoting the given image by defined on = [0,1] x [0,1]. We start by 
assuming that each point x G Q can be approximately expressed as an weighted 
combination of its neighbors (eight neighbors are used in this paper): 



, where x -\- 5i are the neighbors of x, with Si = {—h^—h) etc. For grayscale im- 
ages, for example, the above equality is trivially satisfied for an infinite number 
of combinations of w^'^\i = 1..8, given the image u. So trying to estimate w 
separately for each point x is futile without further constraint. Here we use the 
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variational approach with a TV norm proposed in \7] : 

min Vw^^^\dx (2) 

,where A is a parameter that must be set to balance the fidelity term and the 
smoothness term. This minimization problem can be solved iteratively using 
the corresponding evolution PDE: 

, which can then be discretized and solved numerically. 



2.1 Connections to LLE 

Given a set of N vectors {Xi] in high dimensions. The LLE solves the following 
minimization problem: 

mmY,{Xi-Y.W.,X,f (4) 

i 3 

with the constraint 

Y,W,,=l,Wij>0 (5) 
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where the index j runs over those of Xj that is among the k nearest neighbors 
of Xi. with nearest neighbors determined by some metric (i(Xi,Xj). This 
minimization problem has a non-trivial solution since it is usually assumed that 
the dimension of Xi is much bigger than k. 

To generalize LLE, we first assume that the data come in with two com- 
ponents Xi = {Yi^Zi) (think of Yi as grid position, and Zi as intensity value). 
Now we can minimize the following: 

mmY,{Zi-Y.W.,Z,f (6) 

* 3 

the index j still runs over k nearest neighbors of Xi . but now with nearest neigh- 
bors determined by some metric d{Yi^ Yj) depending on the other component of 
X. 

If dimension of Xi is small compared to k (as in the case of an image), 
we must add regular izat ion term to make the problem well-posed. And we will 
recover the discrete counterpart of ([2]) after ignoring the convexity constraint ([5]). 



2.2 Comparison with Digital TV Filter 

The digital TV filter also implicitly estimates some kind of local linear struc- 
ture. It repeatedly applies the following filtering step on the current image until 
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(a) k;*^*^ computed in our method 




(b) /i^*^ for TV filter after convergence 

Figure 1: The linear coefficients computed in our method and digital TV filter 
is quite different for a simple image 

convergence ( see equation (9) in [Ij): 

8 

u'^\x) =^h^'\x)u\x^6,)^h^''\x)u\x) (7) 

, where is the reconstructed image at the t-th iteration, and the coeffi- 
cients h are based on local image gradient and must be recomputed for each 



4 



iteration. Note one difference in form between ([T]) and ([7j) is that we don't have 
a corresponding w'^^^ in ([T]). The reason is that if w'^^^ were added, ([2]) would 
have a trivial solution, with w^^"^ = 1, w^'^'^ = 0,z = 1 . . .8. The coefficients for 
digital TV filter after convergence is quite different from the coefficients esti- 
mated from our method (Fig. 1). This is due to the fact that we estimate the 
coefficients using a global functional, and the TV filter computes the coefficients 
using local information only. Another reason for the difference is that our ap- 
proach estimates orientation- specific weight, the digital TV only estimates local 
gradient magnitude. 

3 super-resolution 

Suppose now we are given an n x n low-resolution image ix^^^, and have also 
obtained an initial reconstructed zn x zn high-resolution image uq (by nearest 
neighbor interpolation, for example), where z is the magnification ratio. Using 
variational method outlined in the last section on u^^'^ , we get eight nxn weight 
matrices k;^*^ representing roughly the local structure in the low-resolution im- 
age. These weight matrices cannot be directly used as filters for high-resolution 
image because it is smaller in size. To obtain suitable zn x zn weight matrices 
w^^\ we can use linear interpolation on the low-resolution weight matrices to 
obtain the high-resolution weight matrices. Finally, the weight matrices can act 
as adaptive filters for interpolation: 



4 experimental result 

We compare the proposed method with both bicubic interpolation and digital 
TV filter using grayscale images. We use z = 3 in the following experiments. 
The smoothing parameter used is A = 0.1 in all our experiments. It is observed 
that the result is quite similar for all different smoothing parameters we tried. 
The reason is probably that the smoothness parameter affect the smoothness of 
the weights, and only has indirect effect on image itself through the weights. We 
first apply the algorithm to the house image (Fig. 2). Both bicubic interpolation 
and digital TV filter oversmooth the image. To make this point clearer, the 
Sobel edge detector is applied on the results using the same threshold. The 
edge detector returns less edges on both bicubic interpolated and TV filtered 
images, which shows the oversmoothing problem of both methods. Another 
advantage of our method over bicubic interpolation is that the edges are less 
wiggled, as can be seen from the roof of the house image in (c) of Fig 2, for 
example. 

As a second example, the fingerprint image is used to see how well our 
method works for texture images. It is well known that TV filter does not work 
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on texture images as can be seen from Fig 3. Our method also gives reasonable 
result in this case. 

5 Conclusion 

In this work, we proposed a new algorithm for single-image super-resolution 
problem using variational method. Instead of working on the image space as in 
the previous work utilizing variational method, we use variational formulation 
to estimate the local structure of an image. The resulting adaptive filter reflects 
both local pixel variance and global image information. The experimental result 
shows some advantage of our method over some previous approaches. A future 
research direction might be to explore other applications of the variational es- 
timation of the local image structure. 
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(a) interpolation results 




(b) response to Sobel edge detector 




(c) zoom-in of part in (b) 

Figure 2: Comparison of interpolation results. Upper Left: original high- 
resolution image. Upper Right: bicubic^interpolation, PSNR=26.9. Lower Left: 
our method, PSNR=27.6. Lower Right: digital TV filter, PSNR=26.9. 



(a) interpolation results 




(b) response to Sobel edge detector 

Figure 3: Comparison of results on fingerprint image. Upper Left: original 
high-resolution image. Upper Right: bicubic interpolation, PSNR=16.3. Lower 
Left: our method, PSNR=16.8. Lower Right: digital TV filter, PSNR=16.2. 
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